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Background / Context: Description of prior research and its intellectual context. 

Learning the meanings of new words is an essential component of early reading development 
(Roskos et al., 2008). Vocabulary is at the heart of oral language comprehension and sets the 
foundation for domain-specific knowledge and later reading comprehension (Beck & McKeown, 
2007; Snow, Burns, & Griffin, 1998). As Hart and Risley report (1995), vocabulary at age 3 is 
strongly associated with reading comprehension at the end of grade 3, and predicts the trajectory 
of word acquisition that impacts future academic learning. It is well established, however, that 
there are significant differences in vocabulary knowledge among children from different 
socioeconomic groups beginning in young toddlerhood through high school (Hart & Risley, 
1995; Hoff, 2003). Extrapolating to the first four years of life, Hart and Risley (2003) estimate 
that the average child from a professional family would likely be exposed to an accumulated 
experience of about 42 million words compared to 13 million for the child from a poor family. 
Farkas and Beron (2004) in a recent analysis of the children of the National Longitudinal Survey 
of Youth 1979 cohort (NLYDY79), found that more than half of the social class effect on early 
oral language was attributable to the years before five, and that rates of vocabulary growth 
declined for each subsequent age period. Further, children from low-income groups tend to build 
their vocabulary at slower rates than children from high SES groups (Anderson & Nagy, 1992), 
potentially creating a cumulative disadvantage over time. 

In addition, our previous meta-analysis on the effects of intervention on children’s word learning 
(Marulis & Neuman, 2010) has shown that, while vocabulary interventions are effective in 
increasing children's word learning (overall effect size of .88), they are significantly less 
effective for low income at-risk children (g=. 77, SE=.12 Cl95=.53, 1.01) than for middle to high 
SES (g=1.35, SE=.26, Cl95=.85, 1.85), Qb(l)=4.19, p<.05. To date, the previous studies and 
meta-analyses (e.g., Mol, Bus, deJong & Smeets, 2008; Mol, Bus & deJong, 2009; Elleman, 
Lindo, Morphy, & Compton, 2009; Marulis & Neuman, 2010) have provided little infonnation 
about the effectiveness of training on the improvement of at-risk children’s early receptive and 
expressive vocabulary. This meta-analysis is designed to build on the existing knowledge base 
by examining vocabulary interventions specifically for factors associated with child outcomes 
for at-risk children. 

Purpose / Objective / Research Question / Focus of Study: Description of the focus of the research. 
Our goal was to examine not only how instructional practices affect child outcomes for at-risk 
children but also whether the effects of instruction differentially affect various types of at-risk 
children. Specifically, we addressed the following questions: 1) To what extent are vocabulary 
interventions an effective method for at-risk children prior to conventional reading instruction? 
2) What factors (e.g., pedagogical, methodological, intervention, or design) are associated with 
significant word learning gains for children at-risk? 3) (How) are various risk populations (e.g., 
SES, ELL, low academic achievement, low vocabulary pretest scores) differentially affected? 4) 
How do cumulative risk factors affect children’s vocabulary gains? 

Setting: Description of the research location. 

*N/A (meta-analysis) 

Population / Participants / Subjects: Description of the participants in the study: who, how many, key 
features or characteristics. 

*N/A (meta-analysis) 
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Intervention / Program / Practice: Description of the intervention, program or practice, including details 
of administration and duration. 

*N/A (meta-analysis) 

Research Design: Description of research design (e.g., qualitative case study, quasi-experimental design, 
secondary analysis, analytic essay, randomized field trial). 

*Meta-analytic review (see below) 

Data Collection and Analysis: Description of the methods for collecting and analyzing data. 

Study Inclusion Criteria. In order to be included in our meta-analysis, studies had to meet the 
following criteria: a) Study participants were aged 0-6.0 (approximately birth through 
kindergarten) and were free of developmental or neurological impairments such as Down's 
syndrome, William's Syndrome or Cerebral Palsy but qualified as 'at-risk' such that 90% or more 
of the sample were from low-SES families, marginalized racial groups (e.g., African-American 
and Hispanic) or urban/rural communities, were English language learners (ELL), had language 
impairments, low previous vocabulary scores or low academic achievement; b) The study 
included a training, intervention or specific teaching technique to increase word learning 
(defined as receptive or expressive language); c) A (quasi-) experimental design was applied 
incorporating one or more of the following: a randomized controlled trial, a pretest-intervention- 
posttest with a control group, or a post-intervention comparison between pre-existing groups 
(e.g., two kindergarten classrooms); d) The study was conducted with English words, excluding 
foreign language or nonsense words; e) Outcome variables included a measurement of word 
learning, identified as either expressive or receptive vocabulary development or both. 

Study Search and Retrieval. We searched the following electronic databases for published and 
unpublished studies: PsycINFO, ISI Web of Science, Education Abstracts ProQuest 
Dissertations & Theses and Educational Resources Information Center (ERIC CSA & OCLC 
FirstSearch) through September 22, 2008 using the following search term: Word learning OR 
Vocabulary AND intervention OR instruction OR training OR building OR experience OR 
learning OR development OR teaching. In order to maintain and code our library of citations, we 
imported them into the Bibliographic program Endnote. Using preliminary coding, 3,548 
citations were deemed potentially relevant and subsequently retrieved and read in full. In 
addition to the electronic search, we contacted experts and authors in the field for any published 
and unpublished data (their own or that of colleagues) and relevant references. We received a 
32% response rate, generating 36 manuscripts. Therefore, through both electronic search, 
manual search and author communication, we attained a total of 3,584 papers. 

Inclusion Coding. Four University graduate students received extensive training in both general 
meta-analysis coding procedures and procedures specific to our vocabulary meta-analysis. 
Subsequently, a training set of 50 studies was coded separately by all coders. The level of 
agreement reached between the four raters on their inclusion detennination (Fleiss' Kappa = .96) 
falls well within the “almost perfect agreement” range (Landis & Koch , 1977). Thus, each 
coder individually coded the remainder of the studies. Thirty-six papers met all five criteria. 
Study Characteristics/ Potential Moderators. We also coded the characteristics of each study 
and intervention that we believed, based on past research and theory (e.g., NPR 2000, Mol et al, 
2008 & 2009; Elleman et al, 2009, Marulis & Neuman, 2010; Coyne et al, in press) would 
influence child outcomes for at-risk children as well as all information related to participant risk 
factors. Due to the large number of variables and importance of accuracy, training was conducted 
for 6 hours per week over a 8 week period by the first author and involved tutorials on research 
design, variable coding and practical coding techniques. In addition, the first author created a 
coding sheet with accompanying coding manual. At the conclusion of the training, all four 
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coders coded 5 papers in full together with extensive discussion and revision of the coding 
manual and sheet in accordance to the sample studies. Next, the coders coded 5 more papers 
independently and met to compare and discuss. Fleiss' kappa was calculated for the four coders 
at .67, which, though falling within the "substantial agreement" range was not sufficiently high 
enough to allow for proper use of moderator analysis. Borenstein, Hedges, Higgins, and 
Rothstein (2009) and Lipsey and Wilson (2001) recommend an agreement level of at least .81. 
Therefore, we initiated a second round of coding and revisions to the coding sheets. We 
independently coded an additional 35 papers (64 studies; 150 effect sizes) and achieved an 
agreement level within the “almost perfect agreement” range (k=.89). Studies were then coded 
individually by one of the four trained coders. 

Effect Size Calculation. To calculate effect size estimates, we entered the data into the statistical 
program Comprehensive Meta-Analysis (CMA; Borenstein, Hedges, Higgins, & Rothstein, 2005) 
and standardized by the change score standard deviations (SDs). All effect sizes were expressed 
as Hedges’ g. We then weighted the effect sizes by the inverse of their error variances in order to 
factor in the proportionate reliability of each one to the overall analysis (Shadish & Haddock, 
1994). In order to avoid dependency in our effect size data (e.g., when a study used more than 
one outcome measure or treatment group resulting in multiple effect sizes per study), we used the 
mean effect size for each study across conditions while not pooling the variable of interest in 
conducting the moderator analyses (Borenstein et al., 2009; Cooper & Hedges, 1994). 
Publication Bias and Outliers. We examined publication bias using the classic fail-safe N test, 
which indicated that we would need to be missing 5,538 studies (with null results) to invalidate 
our results. This number far exceeded the criterion number (i.e., 5k+10=205 where k=39 studies, 
(Rosenthal, 1991)). None of our effect sizes qualified as outliers (i.e., 3 standard deviations 
above the sample mean; SD=.62). 

Findings / Results: Description of the main findings with specific details. 

Our sample comprised of 36 individual papers which yielded 39 studies and 1 12 effect sizes 
resulting in an overall effect size of 0.94, SE=0.10 Cl95=0.74, 1.14, p<.01. The effect sizes 
spanned from -.10 to 2.13 as can be seen in Figure 1 and Table 1 (please insert figure 1 and table 
1 here). Our sample was largely heterogeneous: Qwithin (38)=284.46, 7 2 =86.64 thus we conducted 
moderator analyses to explain variance / examine which factors of instruction were related to 
child outcomes. 

Grade/ Age. We found no significant difference between the gains received by kindergarten 
(g= 1.13, SE=.17, CI 95 =.80, 1.46) or preschool (g=.80, SE=.13, CI 9 5=.56, 1.05) at-risk children, 
Qb(l)=2.46, p=.12. However, the meta-regression for age showed a significant slope; for every 
month in age, there was an increase of .03 in the effect size; p=.03, SE=.01, Cl95=.008, .05, 
Z=2.71, p=.007. See Figure 2 (please insert figure 2 here). 

Risk Factors. We conducted moderator analyses for seven risk factors: low-SES, marginalized 
race, type of community, ELL, language impairments, low vocabulary pretest scores and low 
academic achievement. For details, see Table 2 (please insert table 2 here). The only significant 
difference found for type of risk factor was SES. At-risk children with low-SES status (g=.80, 
SE=. 12 Cl95=.58, 1 .03) received gains that were significantly lower than middle to high SES at- 
risk children (g=1.50, SE=.25, CI 9 5=1.01, 1.98), Qb(l)=6.32, p=.01; Middle to high SES children 
who had at least one risk factor gained more than low-SES children with at least one additional 
risk factor. 1 These results suggest that poverty was the most serious risk factor affecting child 
outcomes rather than additional risk factors compounding the disadvantage. 

Intervention Characteristics. As can be seen in Table 3, we also conducted eight planned 
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moderator analyses related to intervention characteristics (please insert table 3 here). 

Dosage 

Frequency . There was no difference in effect sizes for studies with the median 30 sessions or 
less (g=1.00, SE=. 21, Cl95=.59, 1.42) than those with more than 30 sessions (g=. 94, SE=.18, 
Cl95=.58, 1.30), Qb(l)=.05, p=.83. As this was a continuous variable, we conducted a meta- 
regression and found that the number of sessions was not significantly associated with effect 
size; [3=-. 004, SE=.002, 095=-. 01, .001, Z=-1.68, p=.09.See Figure 3 (please insert figure 3 here). 
Duration. There was no significant difference between studies with shorter durations than the 
median 56 days (g=.82, SE=.17, Cl95=.48, 1.16) and durations longer than the median (g=1.03, 
SE=.14, 095=. 76, 1.30), Qb(l)=.92, p=.34. Also, the slope was not significant for the meta- 
regression; P=-.001, SE=.001, Cl95=-.003,.002, Z=-.55, p=.58; Duration of intervention was not 
significantly associated with effect size. See Figure 4 (please insert figure 4 here). 

Intensity. Again, there was no significant difference between interventions lasting more (g=l .12, 
SE=.21, CI 95 =.70, 1.53) or less (g=l . 16, SE=.20, Cl95=.76, 1.56) than the median 20 minutes, 
Qb(l)=.03, p=.87 and the meta-regression slope was not significant; [3=-. 003, SE=.01, CI95— 

.03, .02, Z=-.21, p=.84. See Figure 5 (please insert figure 5 here). 

Taken together, our results suggest that dosage of the vocabulary intervention on its own is not a 
mechanism significantly related to child outcomes for young children at risk. 

Group size. There was no significant difference between interventions given individually 
(g=.97, SE=.18, Cl95=.61, 1.33), in small groups of five or less (g=.95, SE=.18, Cl95= .61, 1.30), 
or in large groups of six or more (g=l . 17, SE=.35, Cl95= -49, 1.86), Qb(2)= .32, p=.85. All group 
sizes appeared equally beneficial for at-risk children's word learning gains. This suggests that 
group size is also not a significant mechanism through which instruction affects child outcomes. 
Interveners. Interventions conducted by child care providers who were non-certified/non- 
degreed (g=.23, SE=.l 1, 095=. 02, .44) were significantly less effective than those conducted by 
parents (g=.71, SE=.26, Cl 95 =- 21 , .1.22), experimenters (g=.95, SE=.24, 095 =. 48, 1.42), or 
certified teachers (g=1.25, SE=.19, Cl95=-89, 1.62), Qb(3)=25.91, p<.001. There was no 
significant difference between gains associated with interventions conducted by experiments, 
teachers or parents, Qb(2)=3.06, p=.22. Our results suggest that the intervener may be an 
important mechanism related to child outcomes for young at-risk children as the majority of 
children aged 0-6, including at-risk children, spend a substantial amount of time with child care 
providers (NICHD, 2003) during their early word learning years. 

Type of training. Explicitly taught word learning interventions had significantly higher gains 
(g=l .01, SE=.16, Cl95=.71, 1.32) than implicit/incidental word learning interventions (g=.57, 
SE=.13, Cl95=.33, .82), Qb(l)=4.78, p=.03. In addition, interventions that combined both explicit 
and implicit instruction (g=1.52, SE=.17, , Cl95=.l-18, 1.85) were significantly more effective 
than either explicit alone (g=l .01, SE=.16, Cl95=-71, 1.32), Qb(l)=4.79, p=.03, or implicit alone 
(g=.57, SE=.13, Cl95=.33, .82), Qb(l)=19.83, p<.001. This distinction was useful because it 
allowed us to examine the pedagogical strategy within similarly-identified interventions. For 
example, one study examined implicit word learning through dialogic reading, while another 
intervention used direct instruction of words prior to dialogic reading. Our results suggest that 
the way in which words are taught is an important mechanism for vocabulary growth. 

Target words selected. Studies whose experimenters specifically selected target words prior to 
the intervention (g=1.22, SE=.16, 095=. 90, 1.54) had significantly higher effect sizes than those 
who did not (g=.73, SE=.12, Cl95 =: .49, .97), Qb(l)=5.80, p<.05. Selecting words to be taught 
appears to be a mechanism for greater word learning (on both global and proximal measures). 
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Study Design Characteristics. We conducted two planned moderator analyses for study level 
characteristics. See Table 4 (please insert table 4 here). 

Random assignment to conditions. Studies employing random assignment to conditions 
(g=.53, SE=.15, 095 = 23 , .82) had significantly lower effect sizes than those not employing 
random assignment to conditions (g=1.07, SE=.12, , Cl95 = .84, 1.31), Qb(l)=8.10, p=.004. 

Nature of the control group. There was no significant difference between studies employing 
controls that received nothing (g=.65, SE=.28, 095 =. 10, 1.21), business as usual controls (g=.77, 
SE=.21, Cl95=.36, 1.19), alternate treatment control groups (g=.97, SE=.22, Cl 95 =. 53 , 1.40), or 
within-subjects controls (g=1.32, SE=.23, Cl 95 =. 86 , 1.78) ,Qb(4)= 4.33, p=.23. 

Outcome Measure Characteristics. Lastly, we found that interventions assessed with author- 
created tests (g=1.37, SE=.20, Cl95 = -98, 1.76) had significantly higher effect sizes than those 
assessed with standardized measures (g= .68, SE =.l 1, Cl95=.46, .91), Qb(l)= 8.91, p=.003. See 
Table 5 (please insert table 5 here). The standardized measures may provide a more accurate 
indication of word learning growth for at-risk children as they have a global scope. 

Conclusions: Description of conclusions, recommendations, and limitations based on findings. 

Our results provide strong evidence for the overall effectiveness of vocabulary interventions for 
young, at-risk children. The effect sizes obtained are considered both educationally significant 
(Lipsey & Wilson, 1993) and large (Cohen, 1988). Due to the inclusion of 14 unpublished studies 
(36% of the sample) with significantly lower effect sizes (please insert table 6 here), it is possible 
that our overall effect size is a low estimate. However, the studies using random assignment 
(g=.53) and standardized outcome measures (g=. 68 ) may be a more accurate representation of 
the effects of vocabulary intervention on at-risk children's word learning outcomes. Still, on the 
most conservative end of .53, a moderate word learning gain was obtained. This is tempered by 
the fact that 69% of the studies' participants (who reported vocabulary pretest scores) scored at or 
below the 16th percentile (standard score: 85) at pretest. Thus, even a large gain may not be 
enough to narrow the achievement gap for children with various risk factors. In addition, the 
children living in poverty with at least one other risk factor received the smallest gains and also 
tended to have the lowest baseline scores. In essence, interventions would have to accelerate, 
rather than merely improve, children’s vocabulary development, to narrow the achievement gap. 
Our results indicate that even more powerful interventions than have generally been conducted 
are needed for at-risk children and that the mechanisms through which to achieve this are related 
to the type of intervener, the explicitness of instruction, and whether specific words are targeted 
prior to the intervention. The study that produced the largest effect size (Coyne et al, 2007, Study 
1 ; g=2.13) provided direct teaching of specific words targeted in advance, with 'interactive 
opportunities.. to interact with and discuss target words in varied contexts beyond those offered in 
the story' (p. 77). This exemplified instruction powerful enough to narrow the achievement gap. 
Neither the dosage nor the group size of the intervention were related to the magnitude of the 
child outcome suggesting that policy efforts focus on the quality of instructor and instruction, 
particularly for the at-risk children in poverty who may have the most limited access. 

Altogether, our meta-analysis provides some promising recommendations for classroom settings 
for at-risk children. However, these moderator analyses should not be interpreted as testing 
causal relationships (Cooper, 1998; Viechtbauer, 2007). Rather, our results should be verified 
through experimental manipulations that vary these factors systematically. This research, paired 
with our meta-analyses, would best elucidate the educational mechanisms through which 
practices and policies affect word learning outcomes and differentially affect at-risk children's 
word learning. 
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Appendix B. Tables and Figures 

Not included in page count. 

Table 1. Key characteristics and effect sizes of meta-analyzed studies with at-risk participan ts 
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Note. The symbol - indicates missing, insufficient, or unclear information. 
a Sample was coded at risk if at least 50% of the participant sample was within one risk category: 
low SES (at or below the national poverty level of $22,000); parental education of high school 
graduation or below; qualification for free and reduced lunch; second language status; low 
achievement (as identified by teacher report, achievement or AYP); IEP or Title I placement. 
h AB=Audio Books; AAC=Augmentative and Alternative Communications system; 
BEEP=Bilingual Early Childhood Educational Program; CAI=Computer-assisted instruction; 
DR=Dialogic Reading; IBI=Individual Bilingual Instruction; IR=Interactive Reading; 
LIP=Language Intervention Program; LT=Language Training; LTL=Ladders to Literacy; 
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£ n=Received no treatment (includes wait list); tu=treatment as usual; at=alternate treatment; 
ws=within-subj ect. 



2011 SREE Conference Abstract Template 



B-2 




Table 2. Mean Effect Sizes for risk factors 
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a Qwithin refers to the homogeneity of each subgroup (df = k- 1 ) 
b Qbetween refers to the moderator contrasts (df =number of subgroups- 1) 
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Table 3. Mean Effect Sizes for Characteristics of Interventions 
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“Qwithin refers to the homogeneity of each subgroup (df = k-1) 
h Qbetween refers to the moderator contrasts (df =number of subgroups- 1) 
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Table 4. Mean Effect Sizes for Study Design Characteristics 
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Table 5. Mean Effect Sizes for Outcome Measure Characteristics 
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Table 6. Mean Effect Sizes for 
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“Qwithin refers to the homogeneity of each subgroup (df = k-1) 
h Qbetween refers to the moderator contrasts (df =number of subgroups- 1) 
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Figure 1. Forest plot of effect sizes (k=39). 
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Note. Each circle represents each individual study and is proportionate to its weight in the 
overall analysis: the circles and confidence interval bars illustrate the estimated precision 
of each study. The diamond represents the mean effect size for the entire sample. 
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Figure 2. Meta-Regression of Age in Months. 



Regression of Age in Months on Hedges's g 




Note: The area of each circle is proportional to its weight in the overall analysis. 
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Figure 3. Meta-Regression of Number of Total Intervention Sessions. 



Regression of IV Frequency Regression on Hedges's g 




Note: The area of each circle is proportional to its weight in the overall analysis. 
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Hedges's g 



Figure 4. Meta-Regression of Duration of Intervention. 



Regression of IV Duration Regression on Hedges's g 




Note: The area of each circle is proportional to its weight in the overall analysis. 
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Hedges's g 



Figure 5. Meta-Regression of Length of Each Intervention Session. 



Regression of IV Dose/Intensity Regression on Hedges's g 
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Note: The area of each circle is proportional to its weight in the overall analysis. 



1 In order to determine whether this difference was a function of the number of risk factors present (e.g., middle to 
high SES children having one risk factor compared to more than one risk factor for the low-SES plus an additional 
risk factor), we examined the median and mean number of risk factors for these two groups and found that they were 
quite similar. Both low-SES at-risk children and middle to high SES at-risk children had a median of 4 risk factors. 
The mean number of risk factors was only slightly higher for the low-SES group (4.0) as compared to the middle to 
high SES group (3.4). To examine this further, we conducted a meta-regression on number of risk factors and found 
that number of risk factors was not significantly associated with effect size: P=-.10, SE=,10, Cl95=-.30,.10, Z=-.10, 
p=.34. 
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